Nick Says

I tend to do a bit better with a physical textbook, so I purchased a copy of Compilers: Principles, Techniques, and Tools 2nd ed (i think i somehow ended up with a weird international edition again). One of the exercises in the chaper on lexical analysis was to write a lexer that turns a string into it's pig latin equivalent. For example, 'Big' become 'igBay'. I took this as a chance to figure out SLY and wrote the lexer in python. It works! I do have a question though:

When I originally wrote the program, I had the tokenizing loop in the if __name__ in '__main__' section print the tokens directly. The result looks like this:

Yours-MacBook-Air:programming-languages hypatia$ python3 piglexer.py 
/usr/local/lib/python3.7/site-packages/sly/lex.py:311: FutureWarning: Possible set intersection at position 21
  cpat = cls.regex_module.compile(part, cls.reflags)
/usr/local/lib/python3.7/site-packages/sly/lex.py:327: FutureWarning: Possible set intersection at position 21
  cls._master_re = cls.regex_module.compile('|'.join(parts), cls.reflags)
Token(type='CONSONANT', value='gBayiay', lineno=1, index=0)
Token(type='CONSONANT', value='dbayaay', lineno=1, index=4)
Token(type='VOWEL', value='ppleayaay', lineno=1, index=8)

However, I realized this was kind of messy, so I decided to take the token.value strings and concatenate them into a result, then print that.

Yours-MacBook-Air:programming-languages hypatia$ python3 piglexer.py 
/usr/local/lib/python3.7/site-packages/sly/lex.py:311: FutureWarning: Possible set intersection at position 21
  cpat = cls.regex_module.compile(part, cls.reflags)
/usr/local/lib/python3.7/site-packages/sly/lex.py:327: FutureWarning: Possible set intersection at position 21
  cls._master_re = cls.regex_module.compile('|'.join(parts), cls.reflags)
Input: Big bad apple
Result: igBay adbay appleay 

Strangely, the values printed in the concatenated string are different from the values stored inside the token, even though the only change I made was concatenating the values together into one result. Not really sure what's going on there...

attachments

		last modified	size
	piglexer.py	Sun Apr 28 2024 11:28 am	1.3K